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ABSTRACT 

This document comprises four papers from a symposium 
on using an inqui ry-based observation tool for assessing teaching and 
learning on various educational levels. The first paper, 
"Introduction, and Overview and Use of the Tool for K-12 Science, 
Mathematics, and Technology Program Evaluation" (M. Jean Young), 
presents an overview of the tool including a review of the 
literature, a discussion of how the tool was edited and revised 
through extensive use, and a brief description of training others to 
use the tool. The second paper, "Using an Inquiry-based Observation 
Tool for Assessing Quality of K-8 Staff Development Program in 
Science and Mathematics" (Belle Brett), examines the Teachers' 

Academy for Mathematics and Science in Chicago, an independent 
institution focusing on the professional development of Chicago's 
public schooJ elementary teachers, specifically in the area of 
science and mathematics, and its "Intensive Teacher Enhancement" 
strand. This strand, only one of several designed to promote 
systematic reform in science and mathematics in Chicago's schools, is 
a 3-year inservice program in mathematics and science. The third 
paper "Using an Inquiry-Based Observation Tool to Identify Effective 
Components of a University Course" (Nancy Lemire) , describes a 
project using the Classroom Observation Protocol for Inquiry-based 
Teaching and Learning tool to determine if the behaviors associated 
with inquiry-based teaching and learning were present in a college 
course entitled "Critical Thinking." The fourth paper, "Using an 
Inquiry-based Observation Tool for Benchmarking in a Corporate 
Educational Department" (Susan E. Squires) describes adaptation of 
the protocol for use in evaluating a course for human resources 
specialists. Data gathering forms, along with instructions for 
completing the form and analyzing data, are included. (ND) 
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Introduction, and Overview and Use of the Tool for K-12 Science, 
Mathematics, and Technology Program Evaluation 

M. Jean Young 

National Center for Improving Science Education 

Inquiry-based teaching and leariung is an emerging paradigm at the core of 
many educational reform efforts in the United States and elsewhere. Inquiry- 
based teaching and learning puts learners in the active role, cor\structing their 
own learning, while the teacher acts as facilitator. This learning may take 
place in a variety of modes (e.g., small group hands-on work, whole group 
discussion, one-on-one teacher-student dialogues, individual projects) and 
may take p'ace in many different settings from K-12 schooling to corporate 
education. 

The current wave of reforms in education and education-related settings 
brings with it new calls from the broader community, administrators and 
program managers, goveriunent officials, and funders, for accountability and 
evidence of movement toward inquiry-based teaching and lezuming. 

Assessing the complex and versatile nature of this paradigm requires a new 
methodology that can operationalize its many facets. 

Through our pilot work with a new observation tool for assessing inquiry- 
based teaching and learning, we have demonstrated that we can collect valid 
and reliable information in a variety of settings. The tool is easy to use, and 
non-evaluators can be trained to use it in a minimum amount of time. It is 
adaptable for use in diagnosing areas for improvement, benchmarking, 
documenting evidence of inquiry-based learning, assessing teaching quality, 
and identifying components of inquiry-based teaching that make a difference. 
We believe it is a valuable contribution to the educatior\al and evaluation 
communities and anyone who needs to make observations of these new 
learning modes. 

In this paper, I present an overview of the development of the tool including 
a review of the literature, a discussion of how the tool was edited and revised 
through extensive use, and a brief summary of our experience training others 
to use the tool, which involved developing inter-rater reliability among users 
with divergent backgrounds. Next B. Brett, S. Squires, and N. limire present 
examples of how the instrument has been adapted and is currently being used 
in various settings: assessing quality of K-8 staff development program in 
science and mathematics; identifying effective components of a imiversity 
course; and, benchmarking in a corporate educational department. 
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Development of the Inquiry-Based Observation Tool 

In inquiry-based as well as more traditioruil classes, students may be seen to 
exhibit similar behaviors such as working on laboratory investigations in 
small groups. When inqmry is discussed in the literature intent rather than 
specific kinds of activities or behaviors are generally mentioned. For 
example, the intent of a laboratory investigation in a traditional classroom 
may be for students to follow a procedure in order to answer questions while 
in an inquiry-based classroom the intent may be for students to solve a 
problem. 

Our inqiury-based observation instrument originated in a conversation 
about how evaluators could assess whether or not inqmry was going on in a 
classroom. Specifically, what makes inquiry-based classes look different? The 
particular context being discussed was to show evidence of inq\;^-based 
teaching and learning in classrooms of teachers who had participated in 
science, mathematics, and technology teacher enhancement programs. 

Since an intended outcome for current science, mathematics, and tedmology 
teacher enhancement programs is to move teachers from a more traditional 
approach to more inquiry-based teaching, it seemed reasonable to start 
thinking about an instrument to look at inquiry-based teaching and learning 
in terms of a continuum between two opposite dimensions of beluvior, one 
dimension being "traditional" the other being ’ inquiry-based. Since teachers 
have been using "traditional" methods for decades, the behaviors are well- 
defined. We conducted a literature review to develop our mitial set of 
behaviors for "inquiry-based." We then developed an instrument that ws 
later revised and edited through numerous iterations into its present form 
(see Appendix). 

Need for a New Jnstrument 

The inquiry-based tool was initially developed at the National Center for 
Improving Science Education (NCISE) for use in exploring outcomes of the 
Department of Energy’s Teacher Research Participation (TRP) program. One 
goal of the TRP program calls for teachers to transfer the knowledge gained 
from participation in the program to their students. This knowledge is 
described as "awareness and knowledge of the scientific research process." 
Exactly what an evaluator might look for to find whether or not their 
knowledge was transferred to the classroom was unclear. We needed to 
specify what such terms as "scientific research process" might look like in a 
classroom. 

In developing a classroom observation protocol of which the continuum was 
a part, we first looked at instruments and protocols developed as part of the 
Expert Science Teaching Evaluation Model (Burry, 1992) in particular the 
Science Classroom Observation Record, which was developed to focvis on 
constructivist teaching and learning. We also looked at mstrximents 
developed for the R^orm Up Close study (Porter, Kirst, Osthoff, Smithson, & 
Schneider, 1993). While these instruments as a whole were useful, they did 
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not specifically address our purposes. We were looking for an instrument • 
that would: provide a snapshot of classrooms prior to a program invention to 
conipare widv the same classrooms after the progrcim; look at observable 
behaviors versus making inferences; and be appropriate for use by a variety of 
different observers with diverse backgrounds. Therefore, while we used 
many items and ideas from Burry and from Porter et al., the heart of our tool 
became a typology, a continuum rating scale with more inquiry-based 
practices on one end and more traditional practices on the other. The 
typology allowed us to focus directly on inqxairy-based teaching and learning. 
We refer the reader to the Burry and Porter et al. studies for a review of the 
other parts of the observation tool which relied heavily on their work. The 
following is a discussion of the literature as it relates to the development of 
the typology. 

Literature Review 

Overall, educators agree that science should be taught in schools in ways that 
reflect actual practice of sdentists.(Tobin, Tippins, & Gallard, 1994), i.e., 
scientific inquiry. As Kober (undated, p. 6) says, 

"Young people build critical thinking skills and scientific habits of mind 
when they are allowed to become scientists— rather than simply 
studying science— by modeling the processes of inquiry and exploration 
that real-life scientists use to discover new knowledge."" 

But what does actual practice looks like in the classroom? The National 
Standards in Science Education (May, 1994) presents a comprehensive 
statement of inquiry teaching and learning as: 

"proficiency in conducting inquiry including the use of scientific modes 
of reasoning, and the ability to apply and to communicate scientific 
Imowledge; scientific understanding of concepts, laws, theories and 
models: understanding of the interdependent relationship of science and 
technology; and understanding of the influence of science on societal 
issues both contemporary and historical " 

To operationalize this statement, in order to develop an observation tool, we 
used aspects of a teaching model that is consistent with science and 
technology teaching and learning that incorporates actual scientists' practice 
(National Center for Improving Science Education, 1991). We also used 
specific strategies that are delineated in the National Science Education 
Standards Sampler (May, 1994). 

As a resiilt of the literature review, we were able to pinpoint the separate 
behaviors that distinguish teachers who support inquiry-based tea<^ng and 
learning and resultant student behaviors: 

• The teacher acts as a facilitator by asking students open-ended 

questions, encouraging students to explain and predict in order to 
increase their understanding, and by asking probing questions that 
encourage discussion. 
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• A teacher gives students open-ended problems to solve or questions to 
arjswer through dbing an investigation, 

• Teachers use forms of assessment consistent with inquiry-based 
learning when they test for understanding and ability to inquire/solve 
problems. 

• When students engage in inquiry-based learning, they use reasoning to 
answer questioris or solve problems in order to gain conceptual 
imderstanding and/or explore cause-effect relationships in 
understanding principles. 

• Students may be answering their own questions through experiments 
they have designed and/or through individual and group projects. 

.• Students address one another and often seek help from one another 
versus always looking to the teacher for answers. 

When the teacher does not incorporate inquiry-based learning, students 
generally follow a procedure to solve a problem or follow a procedure to 
confirm, versus explore, a concept or principle through laboratory 
investigations. Assessment is accomplished through short-answer tests that 
seek knowledge of facts and definitions. In a classroom where the teacher 
does not act as a facilitator, students mainly address the teacher, the teacher 
provides knowledge generally through lectures, and recitation versus 
discussion defines student-teacher interactioi\s. 

Initial Use and Revisions of the Tool 

The tool was used in classrooms of ten teachers who were selected to be 
visited before participating in the program and one year later, after 
participating in the program. During the pre-program observations, inter- 
rater reliability' was established between the two observers who made the 
classroom visits. (We are not going to elaborate further on the methods here 
since the current version of the instrument is very different than the version 
initially used.) 

After its initial use, other evaluators were made aware of the instruments’ 
apparent effectiveness as a tool to discern the extent to which inquiry was 
going on in the classroom (see examples that follow this back^oimd section). 
This interest resulted in further developing and revising the instrument and 
establishing its effectiveness for use in contexts other than science, 
mathematics, and technology education. 

During post-program observations of classrooms a year later, we tested out 
several versions of the classroom observation protocol which resulted from 
feedback and input from evaluators using the tool m other contexts. Where 
teachers taught the same subject more than once, different versions of the 
protocol were used separately for the two or more classes <orid later compared 
in terms of quality and quantity of data obtained, i^ter each visit the protocol 
was revised, and then consolidated into other versions. 

In analyzing data obtained using the tool, we were able to see that many of the 
teachers whose classrooms we observed moved toward using more inquiry- 
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based strategies after participating in the program than before participating. • 
Since these outcomes are based on data from a few teachers in our pdot study 
we present them here only as an indication that the instrument has potential 
for use in evaluating teacher programs that promote inquiry-based teaching 
and learning. 

Training Others to Use the Tool 

The current version was developed after training two dozen Department of 
Energy Precollege program staff to use the instrument. The precollege staff 
were trained in the use of the instrument to make classroom visits before and 
after teachers participate in their National Teacher Enhancement Pro^ams 
(NTEP). It was essential during the training to establish inter-rater reliability 
among the staff members because the data collected would be compiled into 
one report from information collected on the nine different programs that 
make up NTEP. 

Using three videotapes supplied to us by Arm Roseberry of Technical 
Education Research Centers (TERC), we were able to have trainees fill out the 
instrument using the same example, discriss their answers, disoiss the 
definitions provided in the instructions to the protocol, then apply their 
knowledge to a new example. After showing the second and third 
videotapes, we determined inter-rater reliability. 

To establish inter-rater reliability we used a system of quadrants whereby we 
divided the typology continuum into eighths then counted responses that fell 
into any given quadrant (two adjacent eighths). We were able to get a 70% or, 
usually, greater inter-rater reliability on all but two items. We then changed 
the delations of these two items to match a shared understanding among 
the trainees. Since the trainees included Ph.D. scientists, teasers, program 
managers, engineers, and school administrators, we felt the instnunent was 
particularly robust in terms of background necessary to use the instrument. 

We would like to note here that the insights and feedback from this group of 
trainees was especially helpful in enabling us to further develop our tool to 
provide observational evidence of inquiry going on in science, mathematics, 
and technology classrooms. 
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Using an Inquiry-Based Observation Tool for Assessing Quality of K-8 
Staff Development Program in Science and Mathematics 

Belle Brett 

National Center for Improving Science Education 

The National Center for Improving Science Education (NCISE) received a 
grant from the Department of Energy to provide an impact assessment of the 
Teachers’ Academy for Mathematics and Science in Chicago (TAMS). TAMS 
is a free-standing institution focusing on the professional development of 
Chicago's public school elementary teachers (K-8), specifically in tihe areas of 
science and mathematics. TAMS "Intensive Teacher Enhancement" strand, 
which is only one of several strands designed to promote systemic reform in 
science and mathematics in Chicago’s schools, is a three-year inservice 
program (one and one-half years each in mathematics, followed by science). It 
includes direct instruction, either at TAMS or on site, and follow-up 
classroom support both during instruction and after. At the time of our 
study, 42 schools had been involved in the intensive progrzun. 

Because both the organization and it programs had been somewhat in flux 
since its inception in 1990, we decided that an impact evaluation would be 
premature. In addition, our ability to look at even short-term teacher and 
student outcomes was limited by our budget and lack of other data needed for 
analysis of TAMS’ teacher participants and their students. Thus, we decided 
to examine the organization’s history and progress and assess the quality of its 
programs. 

One key area for examination was the actual instruction delivered by TAMS’ 
staff to teacher participants. We interviewed aU mathematics staff (the 
director, the three in-house instructors, four "implementation" specialists 
who did follow-up in the schools, and a key consultant who also taught at 
TAMS) and most of the science staff (the coordinator and nine of the twelve 
instructional staff). 

We observed nine classes taught by fourteen different instructors (four classes 
were co-taught by two or more instructors): four mathematics classes (two 
geared towiurds grades K-3 teachers and two towards grades 4-8), four science 
dasses (three aimed at K-3 teachers and one 4-8), and one class offered by 
TAlvIS' Resource Center. In some cases the interviews took place before the 
interviews and in some cases after because of scheduling. 

After considering various possibilities and reviewing the literature, we 
dedded to use an observational protocol to guide our viewing, both in tlie 
instructional dasses and in the schools. Since one of TAMS’ objectives is to 
help teachers develop their skills in using inquiry, we agreed to help pilot test 
an earlier version of the Classroom Observation Protocol for Inquiry-Based 
Teaching and Learning. The Protocol covered several key facets that were in 
aligiunent with TAMS' instructional model. 
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Because we were interested in assessing program quality and ru)t just the 
degree to which certain focused teacher and student behaviors were 
traditional or inquiry-based, we developed an additional tool based on a 
variety of other dassroom observation instruments as well as a staff 
development template incorporating elements of effective practice and 
developed by NOSE. li\e new instrument both built upon and 
complemented the Protocol It covered a broader range of behaviors 
assodated with gocxi teaching(e.g., pacing, darity of explanation) and a 
munerical scale from 1-7 that indicated the degree to which the effective 
behavior was present. Thus, in contrast to the Protocol, there was a "good" 
end of the scale. This instrument was reviewed by several individuals and 
modified slightly after the first set of visits. 

The usual procedxire for use of both protocols was that one person took 
extensive field notes of dialogue whfie the other captured behaviors. The 
observers would try not to communicate with each other in dass sessions and 
often sat in different places to obtain different perspectives. During group 
activities, the observers sat in or observed different groups. At breal^, the 
observers informally chatted with the instructors and/or partidpants. 
Observation time averaged two to two and one-half hours. (Classes lasted 
anywhere from three hours to a full day.) As soon as possible after the 
session, in cases where more than one person observ^, each observer 
completed the scales of the observation instruments. liien both met and 
came up with a third version, a consensus. 

The Classroom Observation Protocol for Inquiry-Based Teaching and 
Learning was completed for seven out of nine observed session of TAMS' 
iristructors teaching teachers. (Therefore, "students" referred to below, are 
TAMS' teacher partidpants.) For three of these— a K-3 sdence class, a K-3 
mathematics class, and a 4-3 mathematics dass— the Classroom Observation 
Protocol was completed by two observers. One of the observers was an 
administrative assistant with NCISE, with no previous dassroom experience 
or backgroxmd in sdence education. A further challenge was presented by 
these dasses than one might find in the usucd dassroom in that they were co- 
taught. There was the danger, therefore, that the two different instructors 
might teach in different ways and thus present confounding data. 
Nevertheless, with minimal training, the inexperienced observer was able to 
achieve high inter-rater reliability for all classes, using the quadrant system 
described previously, with a more experienced observer /evaluator with 
classroom teacliing experience. 

Inter-rater reliability was achieved on all but two items in two dasses, and 
three items in one class. In the latter case, agreement level of two items fell 
within one and one-half quadrants. Greatest confusion arose when behaviors 
were minimally demonstrated or maximally demonstrated. For example, 
with "students ask mainly procedural questions" versus "students ask 
questions to darify conceptual understanding," the student didn't ask many 
questions at all in two dasses. Although they had a great deal to say, it was 
not in the form of questions. Thus, possibly there were r\ot enough data to 
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make a good judgement. Alternatively, the inexperienced observer, who had 
marked towards the end of the scale with "procedural questions" admitted 
that because there was not a lot of data, her tendency was to mark toward the 
left hand side. This type of error in scoring is a possible hazard in bipolar 
scales. 

Conversely, the "teacher talks" versus "student talks" item was problematic 
in that both teachers and students talked a great deal, thus making it harder to 
gauge the actual proportioa As an experienced observer, I find that scripting 
a dass helps me to find the evidence for this kind of question. However, we 
generally do not recommend this procediure for inexperienced observers as 
they are likely to miss a great deal. With two observers, one person can script. 

In the K-3 mathematics dass, there was disagreement about a third item- 
students using versus not using evidence to support daims. In this instance, 
the inexperienced observer felt that her lack of knowledge of the subject 
matter inhibited her judgment in distinguishing the extent to which she felt 
evidence was being presented. On the whole, lack of content knowledge was 
not a problem in noting whether inquiry exists. Fortunately, most people 
have had experience themselves in classrooms to be able to readily pick up on 
the milieu for observational purposes. 

Cond’*^ons under which the observations are made might also influence 
reliability. The K-3 sdence dass took place in a noisy gymnasium and then 
was moved to the library where the intercom and fire bell constantly 
interrupted the dass. This somewhat more chaotic environment was in 
contrast to the controlled conditions of dasses taught at TAMS. New 
observers need to be alerted to factors that might affect their responses. They 
also need to be encouraged to not let one event dominate their perceptions. 

In one instance, the inexperienced observer noticed one unfortunate 
interaction during a small group exerdse between an instructor and a teacher 
partidpant. This observations caused her to score the teacher on one item 
somewhat differently from me. conversely, had I witnessed the inddent I 
might have changed my own assessment on that one item. Awareness of the 
subjectivity of observation ca help to compensate for that very subjectivity. 
We advocate the use of marginal notes to explain responses where the 
observer was xmdear about the appropriate responses. In addition, the use of 
two observers, although not critical, increases the validity and reliability of 
the instrument. We found that by completing the Protocols ourselves first, 
we were each able to score the sessions based on our own perceptions, 
without the undue influence of one observer. Then, each could make 
his/her arguments during the consensus discussion. In all cases, conseixsus 
was easy to achieve— in most cases a score somewhere between the two marks 
was agreed upon, in a couple of cases, an argument was suffidently 
convincing that the new mark was dose to one of the together of fi\e two 
observers. Since our study was qualitative, all data collected were reviewed— 
both of the observers completed the instruments, as well as the consensus 
scales. 
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Fortunately, we found the Classroom Observation Protocol for Inquiry-Based 
Teaching and Leciming to be a fairly low inference tool, in which behavioral 
evidence is paramount. Thus, even with one observer, it can provide 
reliable, quantifiable data, with minimal training and experience, for an 
activity that is often quite subjective. 



Using an Inquiry-Based Observation Tool to Identify Effective 
Components of a University Course 

Nancy Lemire 

Vermont College of Norwich University 



In the spring of 1994, 1 designed a student research project in which I used the 
Classroom Observation Protocol for Inquiry-based Teaching and Learning 
tool. Part of my project was to determine if the behaviors associated with 
inquiry-based teaching and learning were present in a college course entitled 
“Critical Thinking.” 

As part of the University System of New Hampshire, The College for Lifelong 
Learning’s mission is to deliver irmovative learner-oriented programs of 
higher education. One component of this learner-oriented approach is the 
implementation of a core curriculum ^requiring all students to complete the 
Critical Thinking course. 

Although the outcomes for the Critical Thinking course were clearly defined 
by the college, for example, "to view the integrating of skills and the 
synthesizing of ideas as a characteristic of an effective problem-solver, the 
process or strategies for attaining these outcomes were not clear. I needed a 
way to identify the behaviors associated with inquiry-based teaching and 
learning in order to determine evidence of these behaviors in the classroom 
and if these behaviors were related to the proposed outcomes for the course. 

The classroom observation tool designed by NCISE was easily adapted for my 
purposes. As I familiarized myself with the tool, I foimd the behaviors listed 
on the typology were consistent with those in the literature regarding inquiry- 
based teaching and learning. I operationalized definition for the typology 
using the definitions provided in conjunction with my own definition. For 
example, in the section related to student behavior, "students ask mainly 
procedural questions" versus "students ask questions to to clarify conceptual 
understanding," I wanted to be more specific regarding who the students were 
directing questions to. I added "of teacher" to each of these behavior 
definitions, 

I used all sections of the tool except for the section designed for observing 
laboratory or hands-on learning. This section was not used because the 
Critical Thinking course did not include these elements. The omission of 
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this section did not affect the data gathering capacity I reqxiired for this project 
in any negative way. Further, I believe that the division of the tool into 
sections has created an instrument adaptable to many classroom 
environments. 

I observed three Critical Thinldng classes each led by a different instructor 
during the spring semester of 1994. I made three observation in each of the 
three classes, one at the beginning of the semester, one at mid-semester, and 
one at the end of the semester. Each observation coitsisted of a time period of 
three and one-half hours (the duration of the class). At the end of tWs time 
period I scored by observations on the continuum provided for each set of 
behaviors or strategies. I used the suggested percent system in scoring my 
observations (i e., score according to the approxiinate percent of time the 
behavior or strategy was observed to occur), and identified the mid-point of 
each line as 50 percent. For example, if the students interacted with each 
other 75 percent of the time, I placed and "X“ halfway between the mid-point 
and the end of the line on the side where that behavior was listed. 

Once all observations were completed. I began the job of interpreting the 
observation sheets. My first task was to look for evidence of the behaviors 
associated with inquiry-based teaching and learning— if they were occurring 
and how often This was accomplished without having to read pages and 
pages of notes. I could look at the scores I gave each pair on the ty^logy and 
calculate approximate percentages of class time these behaviors occtured, or 
did not occur. 

Scoring separate pairs of behaviors/ strategies for each of the three Critical 
Thin ing classes allowed me to compare and contrast patterns of behavior 
between classes. I was also able to determine whether or not changes in 
behavior occurred from the begirming to the end of the semester. 

As a graduate student and a novice at data collection and interpretation, I 
found the design of the tool uncomplicated and easy to use. It delineated the 
behaviors associated with inquiry-based teaching and learning providing me 
with the framework necessary to collect information focused solely on the 
elements of interest. The separate section devoted to discussions allowed to 
compare that particular class activity between classes, and with the general 
classroom behaviors. I found the separate sections devoted to particular 
classroom activities especially useful in interpreting when and where 
inquiry-based behaviors were evident. 

Having used the instrument somewhat early in its development, I was 
somewhat hampered by the lack of clarity and specificity of the definitions for 
each set of behaviors/strategies. In addition, I would have found it helpful to 
have separate sections involving the assessment or evaluation of the 
student's learning while participating in any given class. 
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Using an Inquiry-Based Observation Tool for Benchmarking 
in a Corporate Educational Department 

Susan E. Squires 
Arthur Andersen and Co., SC 

Goal-based training is an emerging variation of inquiry-based teaching and 
learning used by the more innovative corporate education programs. At 
Arthur Andersen's Center for Profession^ Development this style of 
teaching and learning has been used for several years. The role of training 
has shifted from providing information to a focus on skill development in 
problem solving just as there is a shift in emphasis in the workplace toward 
the roie of the worker as problem solver. 

Goal-based training provides a hands-on experience for the learner by creating 
an environment that simulates and actual business situation that might be 
encountered back in the home office. The learner is then provided with a 
business problem and is asked to provide possible solutions. 

Coach/fadlitators and an auray of learrting materials are made available to the 
learners to use as they wish. Actors may also be used to represent business 
clients or office staff. Therefore, as with other inquiry-based teaching and 
learning, the learner takes an active role in constructing their own learning. 

The Classroom Observation Protocol for Inquiry-based Teaching and Learning 
tool was adapted for use in evaluating a course for human resources (HR) 
specialists. The goal of the course was to introduce staff to more proactive, 
inquirv-oriented roles for the HR professional: consultant and strategist. 
Sixteen students participated. They were divided into four groups and sent to 
four separate classrooms where a simulation of a human resource office was 
recreated. The task of the evaluator was to observe the interactions and 
activities in each of the classrooms and document evidence of learning new 
human resource roles (consultant and strategist) as the participants attempted 
to solve a human resource problem. The two new roles encourage the active 
use of inquiry where as the traditional role of the human resource staff, 
providing services, does not. In conducting the evaluation, I had the 
assistance of three other observers who I oriented to the principles behind the 
inquiry-based observation tool. 

Using the inquiry-based observation tool as a foundation, I broke the 
evaluation process into three components: 

• observation and recording; 

• analysis of the occurrences and frequency of inquiry over time; and 

• placement of analyzed observatiorts on ^e inquiry-based observation 
tool (the typology). 

To record initial observations an open-ended tool was created that focused on 
the items listed in the typology (see below, Human Resources in Our Business: 
Some Things to Look For and, attached Human Resources in Our Business: 
Observation Record Sheet). I had some concern that the other observers, with 
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whom I was working, had a minimal amount of training to use the tool and 
might leap to conclusions about what was happening in the classroom. The 
open-ended tool was designed to record actual observations rather than 
record an interpretation of the observation. Even with this precaution I did 
find that one observer tended to interpret observatioris. 

Once the observations were recorded these documents were analyzed 
evidence of inquiry and non-inquiry behaviors/activities and the frequency 
of occurrence of each was counted (see attached. Human Resources in Our 
Business: Day 2). 

Finally, the types and frequency of inquiry and non-inquiry evidence was 
compared over time to demonstrate increase in inquiry-based learning 
among participants (see the attached Final Results di^rt). 

Findings indicated an increase in staff ability to use inquiry and by definition 
take on the new HR roles providing evidence to the course designers that 
their goals for the course had been successfully met. 



HUMAN RESOURCES IN OUR BUSINESS: Some Things to Look For 



Coach seeks facts 
Participant seeks facts 
Coach begins interaction 
Coach talks /participants listen 
Participants interact with coach 
Coach uses materials as a resource 
Participants use coach as a resource 



Coach seeks understanding 
Participant seeks understanding 
Participant begins interaction 
Participants talks/coach listens 
Participants interact with each other 
Participants use materials as a resource 
Participants use each other as a resource 
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HUMAN RESOURCES IN OUR BUSINESS: Day 1 
Observation Summary of Participant Role Playing 



Service Provider Role Consultant Role Strategist Role 



Tasks/ 

Outcomes 


Each participant kept 
to own work, there is 
no real 

communication 
among them 




Group began to 
brainstorm strategy for 
approaching from 
memo 




5 OBSERVATIONS 


NO OBSERVATIONS 


2 OBSERVATIONS 


Relationship 


1 participant asks for 
HR information from 
EA (files) 

4 participants in 
group meeting asked 
factual questions 




Coach steered recntiting 
person into prioritizing 
and looking at different 
angles to solve problem. 

Participant sought out 
another HR participant 
to get guidance on how 
they would solve 
problem. 




5 OBSERVATIONS 


NO OBSERVATIONS 


2 OBSERVATIONS 


Process 


Director is scheduling 
a meeting with 
counterpart, but does 
not include any other 
of her team 


Team discussed three 
possible solutions to 
the merger problem 


Team identified the 
impact on the structure 
this merger will bring 




2 OBSERVATIONS 


2 OBSERVATIONS 


1 OBSERVATION 
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Final Results 



long 



Obsen'ation 
Summary of 
Firtidpaat 
Roka 

MONDAY 
TUESDAY 



WEDNESDAY 

THURSDAY 

FRIDAY 




KEY nNDINGS 



Participant Roles by "Day* 




Tbere were 124 separate observational evems recorded over tbe 
duration of this course. Using the indicators provided in the 
HRB Observation Grade, the role ofpartiapants ddiing each 
event was classified as either Service Provider. Consultant, 
artd/or Strate^pst. 

Findiogs from the observational data udicates that over the time 
of this course participants inaeasingly engaged problem-solving 
and inquiry behavior that is reflective of the target roles of 
Consuiunt and Strategist. 



On the first roornmg pardcipants ptimaiily engaged b the role of 
Service P . ovidcr. From "Tuesday* to "Thursday" new roles were 
explored. 

By the end of the course a balance between the three HR rotes 
was achieved. Participants were denunstratbg evidence of using 
bquiry/problcm-solving skills. 



I 



I 
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INSTRUCTIONS FOR FILLING OUT THE CLASSROOM OBSERVATION PROTOCOL 



Try to schedule your visit to comcide with the main purpose for your visit. For 
example, our experience has shown that to ascertain the extent to which inquiry is part 
of the learning process, observing during a discussion of a previously-conducted 
experiment or hands-on exercise provides the best data. 

During your visit collect any worksheets, lab sheets, other hand-outs or work 
associated with the lesson. 

PRE OBSERVATION DATA 

If possible, try to fill this out prior to observing classes. 

Qass period or time of class: 

Note ^e time and/or class period(s) you will be observing classes. Ask the teacher 
how often s/he teaches science and to what extent science is a regular part of the day or 
week. 

Topic or topics: 

Ask the teacher to teU you the names of the topic or topics that will be addressed in the 
class(es) you will be observing. 

Placement of class or lesson within the unit of study: 

Note whether the topics taught are somewhat in the begirming, middle, or end of the 
unit of study. 

Placement of class or lesson with the NCISE teaching model: 

Note which stage the class or lesson represents. Note also the extent to which the 
teacher reports s/he is cognizant of eind/or follows the model. 

Purpose (objectives): 

Ask the teacher to list the objectives for all the classes you will be observing. These 
may or may not be the same as "Intended Outcomes" below. Often teachers will state 
some formal objectives for "P;irpose" then tell you what he or she expects students to 
get out of the lesson in "Intended Outcomes." 

Intended outcomes: 

(See above) 

Materials Used (teacher-made, manufactured, district or department-developed?): 

Ask the teacher to describe what materials he or she will be using to teach the lesson. 
These materials may include anything from textbooks to overheads to worksheets to a 
computer program. Be sure to try to get a copy of any matericds used such as student 
worksheets or reports. 

The materials can be chmacterized in a variety of ways. Record here the extent to 
which the materials may support the program goals (e.g.. Do the materials promote 
development of thinking sl^ls?) 
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(Page 2, Notes on filling out the classroom observation form) 



How students will be assessed (for this lesson): 

Ask the teacher what methods are plarmed, and try to get a copy of any assessment 
tools/ instruments. 



CLASSRCX)M ACTIVrnES 

As you are observing the class, take notes on what you observe in the appropriate 
boxes. If you like to write extensive notes and/or you have auranged for one observer 
to take notes and the other person to fill out the protocol, write on the back of the 
Classroom Activities sheet. 

Introduction to Lesson: 

Describe how the teacher starts the lesson (e.g., gives a content overview, relates the 
content to previous work or to science). While it is assumed the student grouping will 
be whole class, there may be an occasion where it is not. Fill in the amount of time 
(Duration) the teacher introduces the lesson. 

Activity/Task: 

Describe the content and the nature of the lesson or classroom activities including the 
method of teaching, how/if students are grouped/interacting. It is particularly 
important to note if the lesson/ activities relate to what the teacher experienced over 
the summer (process as well as content). Describe what the students are doing e.g., 
listening and taking notes, writing answers to questions. Describe how the teacher is 
interacting with the students, and how the students are interacting with one another. 

If several activities are occurring at the same time indicate so at the bottom of the page. 
If more room is needed for notes, write on the back of the sheet. 



OTHER OBSERVATIONAL DATA 

Fill this out during the classroom observation. 

1 - Description of the classroom: 

Describe how the seating is arranged, number and kind of windows and lights, 
describe/list any special equipment or materials. Note especially if there are separate 
areas for different activities (e.g., a "library" with a place for students to sit). Describe 
what is on the walls, especially bulletin board displays. Give an overall general 
description of the size of the room, e.g., 'large' is sufficient. 

2 - Teaching aids/materials (per activity/task if appropriate): 

All materials including chalkboard, overhead projector, teacher-made handouts, 
tex'rbook, should be listed. 

(Pa 9 3, Notes on fil l ing out the classroom observation form) 
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(Page 3, Notes on filling out the classroom observation form) 



3 - Assessment strategies used (per activity/task if appropriate): 

If during the observation the teacher uses some form of assessment strategies, record 
them. For example, a teacher may circulate among students doing work in small 
groups and make notations on a checksheet. 



4 - Time not devoted to teaching and nature of non-academic or procedural activity 
(e.g., management, announcements, discipline); description of non-instructional 
event: 

Give approximate percent of time or actual time not directly devoted to instruction 
(teacher instruction, self-instruction, student-to-student instruction). Non- 
instructional time may be a variety of things including stopping to discipline students, 
talking about last nights' ballgame, or listening to aimoimcements over the intercom. 



STUDENT DATA 

Complete items 1 and 2 when appropriate, e.g., you may get the data on number of 
minorities from the teacher after the observation, or, you may make that assessment 
yourself during your observation. 

1 - Number and gender of students; number of minorities/majority: 

Record the total number of students present dialing most of the class (it is expected 
that occasionally a student will enter or leave during the class period). Record the 
number of females and males. Under certain circumstances figuring minorities is not 
always easy, partly because of students with varied racial and ethnic backgrounds. An 
estimation is acceptable. You may want to get this information from the teacher prior 
to the classroom observation. 



2 - Describe the content of a student's journal or notebook for the class. 

Find out from the teacher if the students are expected to keep a journal or notebook. If 
the teachers gives you permission, ask one of the students if you can look at his or her 
notebook. Record the kinds of entries, the number of pages, and espedally note if 
there is any evidence of problem solving, data collection and analysis, self-evaluation 
or other type of critical thinking. 



OPERATIONAL DEFINITIONS FOR STUDENT BEHAVIORS 

The continua include undesirable student behaviors on the left and desirable student 
behaviors on the right. For each indicate the degree or percentage of desirable and 
undesirable behavior. 

• most students off task = 50% or more of the students are not on task for at 
least 50% of the dass period. 
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most students on task = 90 - 100% of students are on task for the entire 
period (100% of the time). 



• students interact with each other around procedural issues = they are 
asking one another such things as, '^hat did he say?" or, "Do we answer 
questions 5 and 6 or just 6?" 

students interact with each other around content issues = students are 
actively interacting around the lesson or topic. In some cases students may 
seem off topic because they are talking about a related issue. Even if the issue is 
not directly related it should be considered as interacting euround content issues. 



students are hesitant to enter into the discussion/activity = students do not 
actively engage in discussion or engage in an activity and are likely only to 
answer direct questions posed by the teacher. You may see body language that 
corroborates their reluctance. 

students actively and enthusiastically participate in the 
discussion/activity = during a discussion, students are probably calling 
out answers and/or engaging one another in some point of discussion such as 
arguing with one another aroxind an issue. During an activity, students are 
actively engaged. 



OPERATIONAL DE: 



atJii 



IONS FOR TYPOLOGY 



The typology is meant to capture, in retrospect, the observer's overall interpretation of 
where the teacher's practice may fall on each of the continua. The items in the left 
colximn are generally more 'traditional' and the items in the right column generally 
reflect more inquiry. No value judgment of the teacher is intended. Value judgements 
should be left to REFLECTIONS AND INTERPRETATIONS. Each item.is intended to 
refer to something that might be transferred from the teacher's participation in the 
program to their classroom practice. Place an 'x' on the spot you feel best indicates 
what you observe for that class/lesson You might think of the line in terms of 
percent, e.g., if the teacher acts like a source of knowledge for 40% of the time and is a 
facilitator 60% of the time put the 'x' to the right of the half-wav point toward 
'facilitator'. ^ ^ 



Write any explanatory notes in the margin or indicate "N/A" if the continuum is not 
applicable to the classroom you observed. For example, the continuum in the 
Discussion section, "teacher helps students reason through the thinking process — 
teacher provides reasoning" is not applicable in cases where there is no attempt to 
bring students' understanding or thiiddng about a subject/idea to a higher le\^l. In 
this case record both 'N/ A' and a brief comment about the nature of the discussion. 
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(Page 5, Notes on filling out the classroom observation form) 



Students: 

• look for correct answer = students do an activity or engage in discussions and 
focus on "getting the correct answer" (as opposed to "seek truth"). 

accept or revise their "hypotheses" based on evidence = students have 
developed some ideas prior to the cxirrent lesson, perhaps through a 
classrooiri activity. This was their prior idea; it may even have been an 
hypothesis they developed. Now, they use new evidence, either direct or 
though a discussion, and revise their idea based on that evidence. 



do not reflect on others' ideas = students do not build on what other 
students say nor refer to what other students might be saying; neither do they 
act on other students' ideas and/or suggestions. 

reflect on others' comments/ideas = students relate to what others' say 
through discussion or taking some action. Students build on what other 
students say but may not directly acknowledge them by name. 



seek information to complete the assigned work = students may ask 
questions about procedure such as asking the teacher or other students if they 
should finish the exercise for homework, or, may ask direct questions about 
how to answer a particular question in order to complete an assigned task. 

seek clarification of conceptual understanding = students ask the teacher or 
other students for explanations and clarifications of the questions asked in 
order to better understand the content. During a discussion a student may 
relate an experience s/he has had related to the topic in order to fit this 
information into ds or her conceptual understanding of the topic. 



Teacher Role: 

• source of knowledge = teacher is the "sage on the stage" and neither seeks 
nor acknowledges student input. The teacher may ask students questions but 
only m order for them to relate facts or content-specific information. ' 

facilitator = teacher seeks input from students and encoL rages students to 
explain, predict, describe, etc. in order to increase their and other students' 
understanding. The teacher will often seek a student's misunderstandings 
and ask other students to offer a better /different explanation, predictiori, etc 
versus correcting a student. In laboratory or hands-on activities, the teacher 
will offer suggestions and/or work with the students to find solutions or 
work out problems. 
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(Page 6, Notes on filling out the classroom observation form) 



questions/comments ask for memory/fact = teacher looks for the correct 
answer around a fact such as asking for a definition. The teaser generally 
asks short answer questions that require memory. 

questions ask for comprehension/opinion = teacher asks probing questions 
and/ or encourages discussion which requires student understanding. 
(Understanding = the student can apply what they know to a new situation by 
explaining, giving examples, predicting, and interpreting.) The teacher 
generally asks questions that require processing, however, the processing may 
not be in the form of a direct question. Look for implicit and well as explicit 
questioning. 



Classroom Activities : 

• algorithms = procedural steps or formula to solve problems and/or answer 
questions. This is most often seen in mathematics classes where students are 
taught to use a specific procedure to solve mathematics problems. In science 
class it is often seen in 'cookbook' laboratory manuals. 

heuristics = use of overall strategies or plan to solve problems and/or answer 
questions. This can be seen wherever students are asked to use critical 
thinkmg skills. (Critical thinking skills include problem solving, evaluation, 
decision-making, deductive and inductive reasoning.) 



abstract = the content may be of academic interest but is not directly related to 
a student's everyday experience. Students usually perceive the content as 
something they must learn in school, and may have to know to pass a test, 
but isn't anything they would have to deal with in their 'real-life.' 

(Note: it is students' perceptions that count, therefore, to make this entry, you 
have to talk with students or base your judgment on something said in class.) 

connected t.i real-world = the content is perceived as relevant to something 
in the student ' lives or to the understanding of something in the real-world. 
It may also be re lated to something that exists in the real-world, such as 
something the teacher experienced at the Lab, but is not directly part of the 
students' experiences. 



prescribed program = shidents/ teacher use(s) the assigned textbook or some 
part of a commercially prepared textbook package such as worksheets. If the 
presmbed program promotes compiling materials, place and 'x' in the 
position that best describes the proportion of prescribed versus compiled. 
Note that the Pre Observation Data sheet has a place for characterizmg the 
materials. 
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(Page 1 , Notes on filling out the classroom observation form) 



compiled = students /teacher use several different kinds of materials such as 
another textbook, books, magazines, audio-visual, computer materials 
ccjmpiled by the teacher. 



Discussions : note whether or not this is more like 'recitation' thaij 'discussion.' 

• closed questions = no matter who talks with whom, the discussing group 
seeks to determine the right answer, which is usually a fact. (Note: the 
"questions" may be implicit. This continuum is meant to capture the overall 
tenor of the discussion as being closed or open.) A typical closed question is, 
"What is 4 X 4?", or "What are the temperature and moisture conditions that 
define a desert?" 

open-ended questions = no matter who talks with whom, members of the 
discussing group are seeking possible explanations/causes/descriptions/ 
understandings. A typical open-ended question is, "What do you think 
might happen if...?", or "If you got a '4' for the answer and I got a '6', why 
might our answers be different?" 



teacher seek's facts = the teacher encourages students to determine 'the' 
answer to a question or 'the' solution to a problem. 

teacher seeks student understanding = the teacher seeks students' 
understandings and misunderstandings, often as a way to determine class, 
and individual progress (perhaps as a form of assessment). 



students do not use evidence to support claims = students give factual 
answers or read facts off a workbook or lab page without further explanation. 

students use evidence to support claims = students provide data or 
collaborating evidence to support what they are saying. For example they 
might say, "I saw that the longer the water was heated the higher the 
temper atxire got which explains that ..." 



teacher talks = amount of time teacher talks during the discussion. 

students talk = amount of time students talk during the discussion. (Note 
also the number of students who are doing the talking.) 
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(Page 8, Notes on filling out the classroom observation form) 



• students talk only to teacher = the 'discussion' may be characterized as more 
of a recitation when the interaction is between teacher and students, 
however, the continuum suggests that there is probably some mixture among 
students talking with the teacher and talking to another. 

students address one another = students turn toward and talk with one 
another without the teacher as a mediator. (Note; this is to be taken literally. 
Students may refer to what one another has said without talking directly to 
that student. This kind of interaction is captured in another continuum.) 



• teacher provides reasoning = teacher may help students understand a 
topic/principle/idea through providing them with the reasoning behind 
what they are telling students. 

teacher helps students reason through thinking process = teacher asks for 
students' reasoning, encouraging them to support and contradict one another 
through discussion. At both ends of this continuum, student imderstanding 
may reach a higher level, but this end of the continuum is intended to 
capture the constructivist approach whereby students are helped in their 
understandings starting from their own perspectives/observations. 



For laboratory/Hands-on/Fieldwork 

• students follow a procedure to answer a question or conduct an 
investigation = this refers to what educators often call "cookbook" 
investigations. 

students answer a question or solve a problem using open-ended 
instructions = this refers to anything that is more inquiry-oriented. 



students take measurements or determine facts to answer questions (one 
answer = the results of the investigation are a series of one right answers 
even though the students may be taking measurements and even collecting 
other data. 

students collect and manipulate data in order to answer questions (several 
possible answers) = there is no one answer but several answers that are 
appropriate because students are collecting and manipulating data related to a 
phenomena. 
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Classroom Observation Protocol 
Pre Observation Data 



Teacher Date 

School Grade/Level 

Observer Lab /Program 

(Fill this out prior to observing classes.) 

Class period or time of class; 

Topic or topics: 

Placement of class or lesson within the unit of study: 

Placement of class or lesson within the NCISE teaching model (1-Invite; 2-Explore, 
Discover, Create; 3-Explanations and Solutions; 4-Take Action): 

Purpose (objectives): 



Intended outcomes: 



Materials Used (teacher-made, manufactured, district or department-developed; 
characterization of materials): 



How students will be assessed (for this lesson): 
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CLASSROOM ACTIVITIES 



Teacher; 

(Fill this out as you are observing classes) 



Date: 



Introduction to Lesson; provides introduction/motivation/"mvitation"; explains activity and 
how it relates to previous lessons; assesses students* prior knowledge 

Student Grouping Duration 

First Activity /Task; Content nature of activity, what students doing, what teacher doing; 
interactions. 



Student Grouping Duration 

Second Activity/Task: Content nature of activity, what students doing, what teacher doing; 
interactions. 



Student Grouping Duration i_ 

Third Activity /Task; Content; nature of activity, what students doing, what teacher doing; 
interactions. 



Student Grouping Duration 

State whether activities are sequential or are different activities/ tasks done at the same time: 
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OTHER OBSERVATIONAL DATA 



Teacher: Date: 

(Fill this out as you are observing classes.) 

1 - Description of the classroom: 



2 - Teaching aids /materials (per activity/ task if appropriate): 



3 - Assessment strategies used (per activity/task if appropriate): 



4 - Time not devoted to teaching and nature of non-academic or procedural activity 
(e.g., management, announcements, discipline); description of non- 
instructional event : 
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STUDENT DATA 



Teacher: Date: 

(Fill this out during/after the classroom observation.) 

1 - Number and gender of students; number of minorities /majority: 

2 - Describe the content of a student's journal or notebook for the class. 



Use pages 3-4 of Instructions for Filling Out the Classroom Observation Protcol for 
operational definitions of student behaviors. 



Student Behaviors: 




most students 


most students 


off task — 


on task 


students interact with 


students interact with 


each other around 


each other around 


non-academic or 


content 


procedural issues 


issues 


students are 


students actively and 


hesitant to enter 


enthusiastically 


into the 


participate in the 


discussion/ activity — 


discussion/activity 
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TYPOLOGY: INQUIRY-BASED TEACHING AND LEARNING 



Teacher: Date: 

(Fill out after the classroom observation.) 

Use pages 4-8 of Instructions for Filling Out the Classroom Observation Protocol for 
operational definitions for typology. 



Students: 






accept or revise 


look for correct 


their "hypotheses" 


answer 


based on evidence 


do not reflect on 


reflect on others' 


others' ideas 


conunents/ideas 


seek information 


seek clarification 


to complete the 


of conceptual 


assigned work 


understanding 



Teacher Role: 




source of 


•s 


knowledge 


facilitator 


questions / comments 


questions/comments 


seek memory/ 


seek comprehension/ 


facts 


opinion 



Classroom Activities: 




oi gonuiXus ■ " 


— " iicuiriativij 



Emphasis : 

abstract connected to real- 

world 



Materials: 




pr6SCTiu6Ci program ■“ 
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TYPOLOGY: INQUIRY-BASED TEACHING AND LEARNING - Page 2 
Teacher: Date: 



-or Discussions 

Amount of Time Observed: 

Percent of students contributing to the discussion: 

closed questions — - — 



teacher seeks 
facts 

students do not 
use evidence to 
support claims 



teacher talks 

students talk 
only to teacher 



teacher provides 
reasoning 



open-ended questions 

teacher seeks student 
understanding 



students use evidence 
to support claims 

students talk 

students talk to 
one another 

teacher helps students 
reason through 
thinking process 



For Laboratory /Hand s-On /Fieldwork 
Amount of Time Observed: 

Grouping (pairs, threes, fours): 

Cooperative /collaborative (yes, no): 

students follow a 
procedure to answer 
a question or conduct 
an investigation 

students take 
measurements or 
determine facts to 
answer questions 
(one answer) 



Part of a project (Yes, no): 



students answer a 
question or solve a 
problem, using open- 
ended instructions 

students collect 
and manipulate data 
in order to answer 
questions (several 
answers possible) 
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REFLECTIONS AND INTERPRETATIONS 
Teacher: Date: 

(Fill this out as soon as possible after the classroom visit) 

1 - Overall, what happened during the classroom observation (e.g., which stage of 
NCISE model was the teacher using and how effective was its implementation)? 



2 - What didn't happen (e.g., students didn't grasp the idea of the lesson)? 

3 - Alternative ways instructor might have handled the lesson/question/situatiom 



4 - Characterize students and their attitudes toward the subject matter and the 
teacher: 



5 - Notable non-verbal behavior: 



6 - 



S^rises/concerns, especially related to the program goals (e.g., the teacher 
didn't appear to be using the science immersion method): 
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